Keeping the TPC Relevant!

The Transaction Processing Performance Council (TPC) is a nonprofit organization founded in 1988 to define transaction processing and database benchmarks. Since then, the TPC has played a crucial role in providing the industry with relevant standards for total system performance, price-performance, and energy-efficiency comparisons. TPC benchmarks are widely used by database researchers and academia. Historically known for database-centric standards, the TPC has developed a benchmark for virtualization and is currently developing a multisource data integration benchmark. The technology landscape is changing at a rapid pace, challenging industry experts and researchers to develop innovative techniques for evaluating, measuring, and characterizing the performance of modern application systems. The Technology Conference series on Performance Evaluation and Benchmarking (TPCTC), introduced in 2009, and the new TPC-Express initiatives are steps taken by the TPC to be relevant in the coming years and beyond. 1. INDUSTRY-STANDARD BENCHMARKS Industry-standard benchmarks have played, and continue to play, a crucial role in the advancement of the computing industry. Without them we would regress to the situation of the mid-1980s, when the lack of standard database benchmarks led many system vendors to practice what is now referred to as “benchmarketing”— performance claims based on self-designed, highly biased benchmarks. The goal of publishing results from such tailored benchmarks was to make marketing claims, regardless of the relevant and verifiable technical merits. In essence, these benchmarks were designed as forgone conclusions to fit a preestablished marketing message. Similarly, vendors would create configurations, referred to as “benchmark specials,” that were designed to maximize performance against a specific benchmark, with limited relevance to the real world. 2. STATE OF TPC Over the last quarter century, the TPC has developed an array of benchmark suites [1]: TPC-A and TPC-B are designed to measure the performance of transactional systems which later morphed into TPC-C. TPC also developed TPC-E, another transaction processing benchmark. TPC-D was the first decision support benchmark later split into TPC-H and TPC-R. TPC-DS is another decision support benchmark based on complex schema and queries representing modern decision support systems. TPC-VMS is the industry’s first benchmark for databases in virtualization environments. TPC-Energy and TPC-Pricing are common standards across all benchmark standards. These standards have had a significant impact on the computing industry’s use of industry-standard benchmarks. Vendors use TPC benchmarks to illustrate performance competitiveness for their existing products, and to improve and monitor the performance of their products under development. Many buyers use TPC benchmark results as points of comparison when purchasing new computing systems. TPC benchmarks are widely used by database research community; there have been hundreds of publications that referenced TPC benchmarks. While TPC’s contributions have been impressive, they are now challenged by rapid changes in the industry landscape. Longer development cycles are not acceptable. The cloud and big data analytics have become important parts of the enterprise application portfolio. TPC is also experiencing a diminishing number of benchmark publications. There are various reasons for this trend. Foremost, the traditional database and system vendors have consolidated into fewer companies, reducing TPC’s membership. Second, some benchmarks have been manipulated by publishing a very high score using unrealistic configurations. Third, TPC benchmarks have become extremely complicated to develop and run. While TPC’s first benchmarks consisted of a few pages of definition, today’s benchmarks consist of hundreds of pages. Contrary to other standard consortia’s benchmarks, the TPC provides benchmark specifications and not the implementation in terms of executables and tools. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Articles from this volume were invited to present their results at The 39th International Conference on Very Large Data Bases, August 26th 30th 2013, Riva del Garda, Trento, Italy. Proceedings of the VLDB Endowment, Vol. 6, No. 11 Copyright 2013 VLDB Endowment 2150-8097/13/09... $ 10.00.